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Overview 

The Project For Comprehensive Achievement Monitoring (CAM) at the University 
of Itassachusetta, Amherst, proposes to initiate and develop for the Commonwealth 
of Massachusetts an evaluative system which can furnish information useful to 
students and educators alike in adapting study patterns, pacing and sequencing 
of Instruction, developing curricula and, generally, improving the quality of ed- 
ucation. It is widely understood that learning results from one's interaction with 
his total environment. However, the Project has chosen to focus on the variables 
related to classroom instruction for two reasons. First, instruction in the classr 
room or in related learning situations is the core of education. Students learn 
most of their academic skills through Interacting with their teachers, schoolmates 
and the facilities which the school system puts at their disposal. Second, the 
Project has unique strengths in the evaluation of the learnings which are produced 
in this context. These strengths can provide Information particularly useful in 
judging the quality of learning, and suggesting, adaptations which may be deemed 
appropriate in improving it. 

Basic to evaluative technique proposed is the concept of monitoring. As will 
be demonstrated below, earlier methodologies of evaluation suffered from the weak- 
ness of narrow and sparse measurement. CAT. proposes evaluation of continuing 
progress through the use of monitors at frequent and natural intervals in a course. 
This permits feedback to the teachor and the student at regular intervals. It 
facilitates adjustments in Instruction and study which normally would not havs bean 
recognised as needed until a mid-semester or semester were over. 

Bsslc to CAM Is the concept of comprehensiveness. It encourages, Indeed even 
necessitates, a clear and comprehensive formulation of the objectivea of a course 
and avoids the shoals on which many evaluation efforts flounder, i.e., fussy and 




- 2 - 



poorly conceived notions lumped under the rubric of achievement . Further, it 
uses a multi -variable model of school achievement. The variables Include: (a) 

sociological variables dealing with family and social background of the students; 
(b) school related variables such as ADA, expenditures per child, urban-suburban 
character of the school; (c) student variables such as rate of learning and re- 
tention and initial achievement, and personality and vocational Interest profiles. 
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A Comparison of Throe Evaluation Models 

Comprehensive Achievement Koi. oring (CAM) will be described and its 
unique features characterised. After this description two more familiar 
evaluation models (e.g., usual classroom testing! and curriculum project 
evaluation) will also be described and their strengths and limitations men- 
tioned to contrast them with CAM. Finally, the pattern and quality of 
information generated by the models will be compared. 

Description of Comprehensive Achievement Monitoring 

CAM is a system for testing achievement on every objective of a course, 
at frequent test administrations throughout the course. At each test admin- 
istration, performance on objectives not yet taught is pretested, performance 
on objectives just taught is immediately post-tested, and performance on 
objectives taught earlier in the course is measured for retention. Parallel 
test forms, comparable in difficulty and content, are all used at each test 
administration, but each student receives a particular form only once during 
the course. Each form typically has an item for each objective. Each item 
is used on only one test form. The function of a particular item changes in 
relation to the time at which lta objective 1j taupht. Testing may take plsca 
at rsgular intervals (e.g., every two weeks) or st the end of certain instruc- 
tional units. Computer based analyses and reports are made. 
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Speclf 1 cat Ion of objectives . The post fundamental preparatory step for 
the use of CAK Is the specification of the objectives to be evaluated, in 
testable, behavioral terns. Objectives nay be categorized according to numerous 
dimensions, and possibly organized into instructional units. Written objectives 
for a variety of closely related projects or courses may be collated and pooled. 
It is then possible to Identify and select for evaluation those objectives 
which are common to several projects, and those that are unloue to a project. 
Objectives are typically related to achievement; however, CAM is equally suited 
to measuring chanf.es in attitudes or perceptions. The pool of objectives is 
called an objective bank, and a computer prop.ran is available to handle the 
large amount of data Involved. 

Test item s. The second step toward the use of CAM is the construction of 
test items. Every iter, is tied specifically to a single objective, and multiple 
items are constructed for each objective. All items, keyed by objectives, may 
be stored in a computerized item bank, ready for sampling or available for 
revision. 

Construction of test forms . The number of test forms, or monitors , must 
at least equal the number of test administrations planned. Tests ate made 
parallel in content by using the technique of stratified random, sampling. Forms 
are also randomly comparable in difficulty, if an item analysis can be run 
(perhaps on a pretest or an earlier veralon of the course) for Indices of 
difficulty and discrimination, the forms may be made more exactly comparable in 
difficulty. 
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Monltors are intended to be short teste, perhaps ten to thirty it;ema. 

Whether or not a single form covers all objectives for a course is a function 
of the proportion of objectives to itetrs-per-form. It may be necessary to ran- 
domly sample (without replacement) the objectives, before doing the same on 
the test items for each selected objective. This technique of sampling must 
insure that, across forms, all objectlveo are equally represented. The same 
consideration holds when iteras-per-fono exceed the number of objectives ; in 
this case, some objectives may be represented by more than one item on some 
forms. 

Student test groups . Students are divided into test groups in order to 
use all test forms at each administration. Test groups are best constructed 
using random sampling of strata of students based on ability or prior achieve- 
ment in the subject. This assures that each group has a range of students which 
gives representativeness to the data for each test form. 

It is most desirable, for several reasons, to Include every student in 
every test administration, and when set up this way, CAM has been found to be 
a satisfactory substitute for usual classroom testing. However, it io possible 
to use only a sample of the student population, especially if the number invol- 
ved in a project approaches one thousand or more, llany different sampling 
designs are possible. Using the total student population in one test group is 
the design for the conventional project evaluation. Unequal-sited test groups 
may sometimes be an administrative necessity. 

Test administrations . Test administrations may coincide with the completion 
of instructional units, or they may be set at regular intervals throughout the 
course. The latter has advantages in terns of ease of administration, and com- 
parability of results from similar courses taught at diffeient schools. 

A ppended package test s. It is possible to add a section to any monitor, 




and have the results Incorporated with the rest of the CAH data. This feature 
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lcnds flexibility in that, should a epocific diagnostic test seem desirable 
at any point, the data can easily be assimilated. 

Data analysis an d reportin g. Output fro:, the computer programs is as 
follows ; 

For individual students 

After each administration: 

1) total score on that and all previous administrations. 

2) a graphic presentation of the above. 

3) a right-wrong indication for each item on the monitor, coded 
by the objective represented. 

At the end of the course: 

4) average scores, across all monitors taken, on items categorized 
by use into three groups — pretest, immediate post- instruction 
and retention of varying lengths of time. 

For whole group or subgroups (e.g., one classroom; highest and lowest quartlles) 

After each administration- 

1) percent answered correctly out of all items across all monitors, 
for each objective. 

Periodically, as desired (e.g., every 3-5 administrations): 

2) trend data, or achievement profiles, for total score and for 
each objective. 

At the end of the course 

3) same as number 4 under individual students. 

4) item analysis (using, whole group only), treating each item in 
three separate ways, by its three functions--pretest, immediate 
post- instruction, And retention measure. 

Data are analyzed, and reports printed, by computer. Data can be collapsed 

in various ways, to be most useful to students, teachers, project directors, or 

state evaluators. 

Specificity of objectives . Any instruction, no matter hew it to be evalu- 
ated, can call for a high degree of specificity of objectives CA2>, however, 
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rigorously prescribes and requires such specificity. It is the base upon which 
the detailed testing, analysis and feedback of the program rest. 

Specificity of objectives allows similar curricula to pool and match their 
objectives. Uhat is common to all curricula, or to several, is 1'eadily observ- 
able, and provides a meaningful, detailed comparison. Objectives unique to 
individual curricula can pinpoint actual differences concretely and precisely. 

Test items tied to objectives . Each test item iB constructed to measure 
achievement on a particular objective. Therefore, test data always relate to 
definite objectives, rather than aggregates of objectives; this allows evaluation 
procedures to be matched with specific goals of the curricula. In this respect, 

CAM differs significantly from conventional curriculum, project evaluations, where 
standardized materials are used, which have not been closely tied to the specific 
objectives for the curriculum. 

Modification of curricula . Conventional curriculum project evaluation may 
provide some criteria upon which to base one kind of decision about an existing 
project: "drop it" or continue it". These criteria are global rather than re- 

lated to specific contributions of the project. Perhaps one of the most valuable 
characteristics of the comprehensive achievement monitoring model is that it is 
able to provide information upon which to make specific recommendations for re- 
taining strong components of a project, and modifying weak ones. No project is 
as effective as possible, as set up at its inception* therefore, a far core 
pertinent decision about it, now possible with the CAM model, is "drop" or "con- 
tinue with these modifications." 

Data more valid . If there is time on a test for one question for an objective, 
then estimates of group achievement on that objective will be more valid if a 
variety of questions is used across the group, rather than the single question 
typical of both classroom tests and project evaluation. It is important to note 
that the increased validity and comprehensiveness calls for no sacrifice in the 
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econowy of data collection, since each student need still answer only one 
question. 

Pretest of all objectives . All objectives are pretested before any in- 
struction has been given. First, it is important to know whether students 
have already acquired information or skills from outside sources, to that the 
project need not lose students 7 interest by covering material that they can 
handle already. Secondly, an index of effectiveness must ultimately be an 
index related to change in student achievement, attitude or perception. In 
order to document change, it is necessary to have at least two comparable 
measurements of the same characteristic, taken at two different times. 

There is reason to continue pretesting on objectives to be taught later 
in the project, because outside learning experiences, or interaction between 
material taught early in the project and that scheduled to be taught later, 
may both very reasonably cause changes in performance during the project. This 
may lead to alterations, either in the sequence of instruction, or the amount 
of time spent on certain objectives. Uhen the level of achievement rises on 
an objective not yet taught, it may be closely related to material just taught, 
in which case, instruction in the later -scheduled unit could be moved up to take 
pedagogical advantage of the relationship. Another possibility is that, without 
changing the sequence, certain instructional units might be condensed, and the 
pace of instruction stepped up. A single pre-course test, will not provide in- 
formation for making the above decisions. 

Immediate post-instruction test . The usual classroom test covers only 
material just taught. CA IT estimates of group achievement on just -taught object- 
ives are comparable to those available from classroom testing. The number of 
students usually involved in projects makes it possible to test each objective 
with a substantial variety of items, without lengthening any one form of the test. 
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Continual measure of rentention . Since objectives contineu to be tested 
after they have been taught, throughout the rest of the course, there is a con- 
tinual test of retention. Intervals between 1 teach' and "test* times are of 
varying length, and can be matched for precise analysis. For example, it would 
be possible to measure retention spanning appromimately six weeks on all material 
of a course except what is presented during the last month or so. Therefore, 
estimates of achievement can be systematically made for each of the instructional 
units after a specified interval. 

Achievement profile . There are comparable data on achievement for every 
test administration. This makes it possible to plot students' achievement on 
any given objective (or group of objectives) for the entire course. This plot, 
called an achievement profile, gives a graphic presentation of the changes in 
group achievement throughout the course. This achivement profile is a unique 
characteristic of the information available form the CAT! model, and is very use- 
ful in describing and reporting results of course and project research. 

Figure 1 presents hypothetical achivement profiles for five objectives from 
a course. Brief comments below the graph give possible interpretations. It is 
obvious that achievement profiles provide a wealth of information, at whatever 
point in the course they are drawn. On the pretest in the foregoing example, all 
objectives except number 2 shov; achievement at the chance level, or about 20% 
(five-option multiple-choice items). Several decisions could have been made after 
test administration one: 

1) Objective 1 was not learned— reteach it in some other way; 2) Objective 
2 has tested hiph on both the pretest and test administration 1— it would be safe 
to skip instruction in this objective. After test administration 5, two other 
decisions might have been made- 1) Achievement on Objective 3 seems to be slip- 
ping — review is needed, preferably soon 2) Objective 8 seems closely related to 
Objective 5--perhaps it should be taught now instead of later. 
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rapid feedback, could be corrected with 
change in instruction. 

previously known and not taught; without 
pretest, this looks like student learning. 

taught and learned , but forgotten 

well taught 

appears related to objective 5« because 
achievement increases when 5 is taught. 



Figure 1, Hypothetical achievement profiles of group 
achievement on five objectives. 
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Continuous data available . Data are available from every test adminis- 
tration. It is possible to look at group achievement on a single objective, 
groups of objectives, or total content of a course, though this last is generally 
less useful. Data can be summarized in a variety of ways, through the use of 
selected computer programs now available. Desired data are always available 
within a few days for decision-making: it is not necessary to wait weeks or months 
for meaningful analyses. Many evaluation systems are not able to analyze and 
report results with sufficient speed and organization to make the information 
most useful to its recipients. Analyses can be tailor-made for project directors 
or state evaluators. 

One economic advantage of periodic feedback is that a project need not con- 
tinue to its end to discover, after all funds are spent, that the goals of the 
project have not been accomplished. Modifications can be made in the program if 
student performance does not move in the expected direction. 

Description of Usual Classroom Testing 

The usual classroom testing situation includes the following sequence of 
events: first, a set of objectives if specified for a limited instructional 

period, usually from one to four weeks, second, an instructional treatment is 
devised and administered to the students: and lastly, a test at the close of the 
instructional period is administered to measure the extent to which the objectives 
taught during that period have been achieved. 

Students' achievement on material taught during instructional period one is 
tested at test administration one. Achievement for period two is tested at admin- 
istration two, and so ori, throughout the course. 

There is usually a final test' administered at the end of the course, for 
which there may be varying amounts of review offered. Sometimes major tests are 
administered at other times during the course e.g. ju-’.c before report cards are 
issued. 
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Flexible weighting . There is great flexibility in the relative emphasis 
accorded various objectives during the year. Decisions may be made at any time; 
content may be added, dropped or modified. The testing is tailored to the con- 
tent as the course progresses. 

Individual student testing . Usual classroom testing can yield diagnostic 
data on individual student achievement, on the few specific objectives which 
have been taught. 

Tests related to objectives . Usual classroom testing may meet the criterion 
of close relationship between objectives and test items, when the school program 
is defined in behavioral objectives, and the teacher makes some effort to relate 
the items directly to the objectives. 

No pretesting . There is usually no pretest information on students' prior 
achievement on any objective. Teachers usually assume that student achievement 
is due solely to the instruction given them in class. Furthermore, they do not 
know whether learning one objective has affected understanding of another objec- 
tive. Students may also have experiences in other courses, or outside of school, 
either before or during a course, which contribute to their understanding of 
various objectives, whether or not they have been taught yet. 

No test of retentio n. There is no information on students’ retention of 
objectives which have been taught earlier in the school year, except in the event 
of some sort of major test. At that test administration, the interval between 
time of instruction, and time of test-of-retention , is different for every objec- 
tive taught. The interval nay span almost a full school year, or be only a week 
or two. There is seldom any data attached to ouch test results about the date 
of instruction on a given objective. 




- 13 - 



No comparison of student achievement over time . It Is very difficult to 
compare students' achievement from one point in time to another, because at each 
test administration, an entirely different test is used; there is seldom any 
overlap in content, and the overall difficulty can vary enormously from one test 
to another. The only possible comparison of achievement from one time to another 
must use a student’s rani; order in his class. This still leaves no way to examine 
c hanges ir. a total class's achievement over time. 

Description of Curriculum Project Evaluation 

A frequently used strategy for evaluating curriculum projects is to administer 
an extensive achievement test at the conclusion of the project. This may consist 
of a test, or battery of tests, sometimes composed specifically for the project, 
but usually prepared and distributed commercially, e.g. standardized achievement 
tests . 

There is sometimes a pretest administered before the start of the project, 
which is either the same as the post test, or an alternate form of it, but presumes 
to measure the same objectives. 

A single posttest or a pretest-posttest costs less than a more effective and 
complete evaluation system such as CAM. There is a minimum of clerical and admin- 
istrative work needed in actually giving the test, and if a commercially available 
test is used, it may simply be purchased: no staff or time is needed to develop a 
test tailored to the objectives of the curriculum. What little analysis on resultp 
can be done, is relatively easily accomplished. 

Deficient immediate post-instruction testing . In terms of immediate post- 
instruction achievement, the usual curriculum project evaluation measures only the 
objectives taught at the very end of the project in a way similar to usual class- 
room testing (i.e., immediately following the instructional treatment). This means 
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that project directors do not have information on the direct effect of instruc- 
tion immediately after students have been exposed to it. 

Tests of retention . The interval between the teaching of an objective, 
and the end-of -course test-, varies for each objective. Such intervals range 
from a week or two, to a full school year. Therefore* an estimate of achievement 
based only on a posttest is an aggregate of ironediate post- instruction achievement, 
short-term retention, and long term retention. Thin composite score may be made 
up of several subscores, but such subscores still do not indicate much about the 
time interval since instruction. 

No comparison of scores . There is no need to discuss comparability of scores 
from one time tc . ther if the testing is done at only one point in time. Pretest- 
posttest problems are discussed below under sample attrition. 

Test items not specific to objective . In posttests which are designed to 
cover an entire course at only one administration* there is great variation in 
the specificity with which test items have been matched to the objectives of the 
course. This problem is especially apparent when standardized achievement tests 
are used, where general subscores are roughly matched with the stated objectives 
of the project. When only standardized tests and materials are used in a post- 
project evaluation, there is a definite lack of systematic information about the 
achievement on specific objectives in the program. 

I nappropriate weighting . In giving one large posttest, especially a standard- 
ized test, the problem of weighting of objectives presents itself. A variety of 
objectives could be poorly measured while other objectives are heavily emphasized. 

It is likely that the intended pattern of emphasis in the course will not be 
reflected in the evaluation instrument. 

Test not comprehensive . Not only will there be too little emphasis on certain 
objectives, but it is possible that some objectives will not be measured at all. 
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Lack of comprehensiveness in an evaluation technique is a serious shortcoming. 

Problems of sample attrition . All of the above weaknesses in the usual 
curriculum project evaluation design are relatively unimportant when compared 
with the most serious problem of all; the turnover of students. Those students 
who were pretested before the program, and received the early segments of in- 
struction, are simply not there at the time of the posttest. Effectively, this 
reduces the hard data to a posttest on students still enrolled in the project 
during the final week, even if a pretest were administered. Therefore, the 
results may represent very little more than immediate post-instruction testing 
on the objectives taught just before the posttest. Pretest information, if it 
has been gathered, relates only to the incoming abilities of a sample of students 
roughly similar to that available for posttesting. The assumption is made that 
students coming into the project are similar to those leaving it, but the data 
cannot be used statistically in analyzing changes in student achievement, since 
change should only be measured for individual students or identical groups of 
students. 

Comparison of the Pattern and Quality of Information of the Models 

The amount and quality of information available from the three models of 
evaluation described above will serve to summarize the characteristics of each. 

Compar i son of information . CAM yields more information than either the 
usual classroom testing or conventional curriculum project evaluation. The 
pattern of data resulting from each model may be fitted into a matrix, in which 
the rows indicate all the objectives or instructional units of the course, and 
the columns represent the possible test administrations during the entire pro- 
ject. A cell of the matrix which is filled in, represents an estimate of achieve- 
ment for that objective or unit, at that test administration. 

The usual classroom testing pattern is illustrated in Table 1. The diagonal 
line of X's at the last administration indicates a final test, presumably covering 
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all the units of the course. 

Table 2 illustrates graphically the lack of information available from the 
usual pretest--postte8t curriculum project evaluation. This illustration makes 
the assumption, not necessarily well-founded, that a sinple test does in fact 
provide information about every instructional unit. 

It is readily apparent in Table 3 that CAM makes available data on group 
achievement for all of the objectives specified for a course, at each time of 
testing. This comprehensiveness of the data provides the necessary information 
for the variety of purposes discussed earlier in this section. It is easy to 
see how CAM contrasts with the other models of testing, where information is gen 
erally available either on a few of the objectives, or as a composite score for 
all objectives, at a single time. 

Comparison of quality . Table 4 displays seven types of information, and 
estimates their quality as provided by each of the three models. 

Conventional curriculum project evaluation is fair to poor on all of the 
dimensions described. These shortcomings are inherent in the use of single test 
long enough to provide detailed information about student performance on a large 
number of objectives is fatiguing and therefore less valid than short tests. 

One long test excludes systematic pretest, immediate post-instruction, and de- 
tailed retention information. Attrition takes a heavy toll of a pretest sample. 
Feedback is limited to a post mortem on the project's strengths and weaknesses. 

Usual classroom testing provides for the measurement of performance on 
specific objectives on an immediate post-instruction basis. By repeated testing 
the effects of attrition may be minimized. If usual classroom testing data were 
collected across similar projects after similar objectives had been taught, ex- 
tensive information would be available for comparing projects. However, an 
accurate comparison of projects must also include protest and retention in- 
formation. The former is used to adjust for Incoming aptitude and achievement 
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TABLE 4 

Quality of Information 
Available from Three Evaluation Models 



Information 



Evaluation 
speciflo to 
objectives 

Pretest of 
objectives 
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across time 
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profiles 
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Model 

Usual Usual ~ Comprehensive 

Classroom Project Achievement 
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differences in students, and the latter for long-term- retention, or payoff of 
the project. Neither of these is specifically available from classroom testing. 
Feedback occurs frequently during the project, but provides information about 
only one instructional unit at a time. 

Comprehensive Achievement Monitoring provides information for evaluation 
comparable, or superior, to the other evaluation models. Its superiority lies 
in the areas of particular importance to project evaluation: systematic pretests 

and measures of retention of objectives. Feedback can be provided continuously 
and comprehensively so that the projects can be critiqued and adjustments made 
before their end. 

Evaluative Issues 

Comparability . Similarity of instruments and comprehensiveness of the data 
generated arc necessary to obtain indices of effectiveness on a state-wide basis 
or within a school district. It would be difficult to observe change in academic 
achievement if the measures taken at one time, or Jn one school, were not directly 
comparable with measures taken at another tirae or in another school. Further, it 
is of crucial importance in evaluating educational programs that there be a sys- 
tematic, on-going, objective accumulation of information about the achievement 
of all behavioral objectives. Both of these concerns are provided for within 
the structure of CA12. 

Another issue in sharpening the evaluative process and Improving the quality 
of instructional treatment and curriculum design, is that of clearly specifying 
behavioral objectives and performance criteria. It is inherent in the CAM design 
that courses be thoughtfully and systematically planned, without, however, des- 
troying opportunities for creative and ad hoc improvisation. 

Flexibility . Comparability does not necessitate a single standardised 
evaluative instrument as CA3‘. has shown. Tide diversity in course structi..,a must 
be accomodated. What comparability demands is that the objectives of different 
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programs for the same subject be carefully specified and tested. For it is 
Impossible to compare course achievement levels from school to school, or even 
from class to class, if the evaluator is ignorant of the dimensions in which 
they differ. However, carefully specified courses can be compared on their common 
components by the CA1* technique. 
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The Desip, n of the Project 

Scope of the Project 

Purpose . The major goal of the project is the more comprehensive and 
accurate measurement of the quality of education as a basis for the improvement 
of education. Common weaknesses in school curricula such as unneeded redundancy, 
low student achievement, or poor retention, are difficult to identify. However, 
without an awareness of the weaknesses, it is impossible to make specific, con- 
crete corrections. The notions of quality control in manufacturing, where a 
constant sampling of the products identifies irregularities and below standard 
units, closely parallel the concern underlying the project. 

To create an effective monitoring of the quality of education the project 
must develop components for the evaluation. The major conponcnt is available, 
i.e., the methodology and design of an evaluation system. The project proposes 
to provide the staff needed for the evaluation by training state and school 
personnel, and to write the required behaviorally st3ted educational objectives 
as well as the items needed to measure student achievement. 

The demonstration, dissemination, and continuation of Comprehsive Achievement 
Monitoring will be explicitly built into the design of the Project. The use of 
the monitoring by teachers in public schools under the supervision of State 
Department of Education personnel will serve as a demonstration of the technique 
which can be visited easily from any point in the State. The dissemination of the 
technique to other members of the school districts, to other districts in the 
State, and to other organization in the country will be carefully planned as a 
continuous ongoing component in the project. The continuation will be planted as 
a cyclic process to include more teachers and schools in each period of the cycle. 
The professional growth of the participants is a form of permanent benefit which 
will certainly continue. 




Sample Population . The project will involve professionals of several levels. 

To ensure understanding and support for the technique State Education Department 
personnel, school administrators and teachers will be part of the project. Twenty 
schools in a state-wide distribution will be selected as representatives of cate- 
gories like size, organization and wealth. Each school will participate at one 
or two grade levels in upper elementary or high school in mathematics. Approximate- 
ly three hundred students from each school will be involved. 

Operatio n. The operation of the project will include both sequential and 
overlapping activities. The sequential activities will include (a) the development 
of a bank of behavioral objectives and test items, (b) the preparation of training 
materials for the State and school personnel, (c) the offering of workshop exper- 
iences for each type of personnel, (d) the design of CAK tests, (e) the printing 
of tests and objectives, (f) the setting of data processing, and (g) the year- 
long collection, analysis and reporting of data. The overlapping activities would 
be superimposed over the sequential activities. They would be (n) the evaluation 
of the project by questionnaires and interviews and (b) the dissemination of the 
ideas. Each of the activities will be discussed in detail. 

Disseminatio n. For the project to continue after its year of operation, an 
explicit, well-supported effort must be mobilized to disseminate information to 
several audiences. The audiences will be (a) the teachers in other grade levels 
and subjects within the same schools, (b) the personnel at neighboring schools, 
and (c) the staff of other state and national organizations. The dissemination 
process oust be an active, rather than a passive process. Although traditional 
techniques such as journal articles and presentations at professional meetings 
will be used, more active forms of dissemination will be used like (a) creating a 
close neti*ork of information for participating, schools, (b) intra- school workshops 
to promote the operation within the school, (c) invitations to neighboring schools 
for visitations, and (d) invitations to neighboring states to visit. 
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Plan of Operation 

Each of the following sections will discuss in detail different phases of 
the project. The phases are separated into two stages, planning and demonstra- 
tion of CAM in Massachusetts. Dissemination will be a component of both stages. 
The chronology of the phases is displayed in Table 5, Schedule Report Bar Graph 
and Figure 2, Flowchart for Improving Educational Quality through CAM. 

Selection of sample populations . The populations in the project will Include 
state level personnel, school districts, school administrators and teachers, 
academic courses and students. The major criterion for selection will be an 
expressed interest in participating in a CAM demonstration program. The openness 
to learn new ideas in evaluation and to work in an innovative program are of 
primary importance to the large scale demonstrations. 

The state personnel will be selected because of their direct involvement 
with instruction or evaluation in the schools, e.g., Supervisors in Curriculum 
and Instruction, Testing, and Educational Research. Their responsibility to the 
demonstration will be to help in the supervision of the monitoring in one or 
two schools during the year and support the dissemination efforts. They will 
be asked to contribute to the development of materials and to enroll in a work- 
shop on CAM. 

School districts will be chosen as a representative sample of the districts 
in the state. The credibility of the demonstration will be increased. If the 
problems and successes can be clearly shown in an environment close to most of 
the schools in the state. They will be distributed throughout the state and pos- 
sibly follow regional lines. 

From the school districts wishing to participate a sample of about 20 will 




be selected. The school district will commit an administrator, e.g., department 
chairman, assistant principal, or principal, and two teachers to the demonstra- 
tion. These personnel will enroll in a workshop during the stumer on CAM and 
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Figure 2 

Improving Educational Quality 
Through Comprehensive Achievement Monitoring 
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use CA3I during the year. Each school will nonitor from 200 to 300 students in 
one or two courses. 

The benefits to the schools will be directly felt in the training and practice 
of the staff which will prompt valuable professional grov;th in the areas of be- 
havioral objective writing, test iten writing, and CA1S data collection, analysis, 
and interpretation. The materials, e.g. , objectives, items, results and monitor- 
ing, developed in other grade levels and subjects will be available to all schools. 
The in-service promotion and work sessions both within the school district and 
across demonstration schools will serve as a major support of the ideas and means 
of causing their continued use. 

The courses at various elementary and secondary grade levels in mathematics 
will be used in the demonstration. Other subjects on a trial basis may be con- 
sidered. The mathematics curriculum was selected because solid foundation of 
past research has been performed in this area. 

Preparation of objectives an d items . A major component in the evaluation 
of instruction must be a clear definition of the curriculum and of the student's 
performance in each objective. 

To maximize the number of courses, for which objectives and items are 
available, a three step strategy will be followed. First, all of the objectives 
which are available from courses previously monitored by Project CAT will be 
collated along with the items used to measure them. They will form the nucleus 
of the objectives and iters developed for the demonstration. 

Second, the project will use consultants, who are specialists in each 
course, to write a preliminary set of objectives and iters for courses not in- 
cluded in the objectives available and to fill in gaps in the existing materials. 
The materials produced in step one and t M o will be printed and made available to 
the participating teachers during their training workshop. 
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Thirdly, the teachers selected for the project will select, add, and revise 
objectives and items to tailor then to their specific course. The revisions and 
additions made by the teachers will be included in the final version which will 
be duplicated and made available to the participating schools. The schools will, 
therefore, obtain a comprehensive list of possible objectives and items which 
they will be able to use in the future to continue expanding their use of either 
CAM monitoring or performance curricula. 

The Project CAT computer program to store the text of the objectives and 
items may be used to facilitate the selection of objectives and the production of 
the CAM monitors. It hss facilitated the ease of working with testing materials 
needed for Cam monitoring of achievement. The computerized object and item bank 
would allow the dissemination and use of these materials, but have to be keypunched 
which require more resources then typing. 

Preparation of training materials . Persons confident of their ability to use 
CAM monitoring would be the most potent stimulate to the spread and continued use 
of CAM monitoring in the state. Specific training and reference materials must 
be available to the personnel participating in the project. They will be prepared 
to meet the gaols of the project. A training manual or workbook for state level 
and administrative personnel will present excercises in all aspects of the CAM 
monitoring process, i.e., from defining curricula in terms of behavioral objectives 
and performance test items through computer data processing and data interpretation 
of results. Explanations in a step-vise fashion with extensive excercises can give 
the pre-service practices necessary to feel comfortable with the ideas of CAM. 
During the school year, these supervisory will be asked many different questions, 
technical, theoretical, mechanical, and must have a ready reference manual to help 
in answering them. The reference manual will also be prepared. 

For the teaching staff both training and reference manuals are also important, 
but they must address themselves more specifically to the activities involving 
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teachers, students, and parents. Excerclses in curriculum definition and the 
interaction of curriculum goals with instructional activities will be empha- 
sized. Further, the interpretation of computer output for teachers in evalu- 
ating their curriculum and in counseling with individual students. Similar to 
the supervisors the teacher will have a reference manual which can be used to 
answer questions to them. 

A question and answer bulletin will also be written for the students and 
possible for parents. These will provide concise answers to the most frequently 
asked questions. Audio-visual presentations may also be prepared. 

Each of the manuals will go through a pilot test for readability, thorough- 
ness, and usefulness with a sample of individuals from the appropriate populations 
because many training materials have only limited usefulness without adequate 
preliminary trials. 

Training and in-service workshops . Workshops at the end of the school year 
and the beginning of the vacation will train the participants for CAll monitoring 
during the coming year. The workshops will be organized by tho Project CAH otaff. 
They have been divided into sections, first one for supervisors and then another 
for teachers. The reasons for this division are (1) the lower staff requirements 
for two workshops with one half the total number of participants to be trained, 

(2) the supervisors will receive more technical and theoretical training than the 
teachers, and (3) if the supervisors are trained first several can be used as 
trainers for the teachers workshop. 

During the school year supervisors will run intra-school in-service workshops 
for the teachers as a means of discussing problems, presenting more information, 
and disseninating ideas to school staff members not yet using CAll monitoring. 

Also, the project's staff will organize five workshops for all individuals using 
CAll monitoring in the state. The state-wide workshop will be used to disseminate 
information about experiences at other grade levels. 
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CAM monitoring procedures . The details of the CAM monitoring procedure 
have been v;ell worked out. Several modifications in the experimental design 
and computer analysis procedures will tailor the monitoring to the specific 
configuration of schools and of information desired. The discussion will high- 
light the procedures so that realistic estimates of manpower and financial 
requirements can be made. 

The collection of data requires the specification of (a) the source of 
data, (b) the sample of objectives, items and students, (c) the structure of 
the CAM monitors and (d) the schedule of monitoring periods. The students' 
responses to the achievement items will be the major source of data. An effort 
to collect attitude data on a regular schedule will be made. In keeping with 
the model of evaluating instruction, a set of aptitude and demographic infor- 
mation about the students and teachers will be collected. 

The quality and usefulness of the information generated by the CAM monitor- 
ing depends primarily on tho careful sampling of objectives, items and students. 
About one hundred objectives chosen for similar courses at different schools may 
have an overlap in objectives. An overlap would allow a comparison of achieve- 
ment by a similar set of students in different environments. The sequence and 
pacing of the objectives will provide a clear picture of the curriculum goals. 

Items will be available to measure the student achievement of each of the 
objectives. Teachers, curriculum consultants, and Project CAM staff will prepare 
items which will be systematically and further edited and pretested in the field. 
The items and objectives may be stored on computer tape in the objective and item 
bank computer program written by the Project CAT staff. The program would allow 
especially quick and completely accurate distribution of them. 

Each school will determine, in cooperation with the demonstration project 
staff, courses and classes within the course to be monitored. The assignment of 
students to each of the testing schedule groups is one of the procedures used to 
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increase the accuracy of the eatinates of student achievement. The scheduling is 
based upon the background data available about the students and used to maximize 
the number of comparisons which are possible. The resulting assigned schedules 
must be keypunched for the computer. 

The structure of the CAM monitors used in each course mu6t be carefully de- 
signed. They must represent a set of randomly parallel tests which provide the 
type of information needed to accurately evaluate the curriculum and provide 
meaningful information to individual students. The Project CAM staff will be 
responsible for this phase of the project. They will also supervise the printing 
and distribution of the monitors to the schools. The structure of the monitors, 
their correct answers, ar.d the classification of each question in at least four 
dimensions must be coded and keypunched for the computer and the Project CAM staff 
will supervise this phase. 

The schedule of monitoring periods during the year depends upon the overall 
goals of the monitoring. Each school will be considered individually and a schedule 
which will meet the project goals will be developed. The data will be collected 
at each school and forwarded by messenger or mail to the University of Massachu- 
setts for further processing. There v;ill be an attempt at overnight turnaround. 

The organization of the data will follow the procedures shown to be most 
successful in school oriented data processing. The students responses will be 
recorded on answer sheets to be optically scanned and a versatile computer program 
used to develop a data-bank from the CAM monitoring procedure. The computer pro- 
gram edits, collates and records on computer tape all of the data collected about 
students and their responses to test questions. 

The analysis of the data concerns the frequency of the analysis and the type 
of analysis. The frequency of the analysis will depend upon the frequency of 
monitoring. A bi*- or tri- weekly schedule seems most appropriate. In addition 
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to the periodic analysis, computer programs are available to plot graphic repre- 
sentations of the longitudinal data and produce an item analysis which checks the 
reliability of the items. 

The types of analysis ore not limited to those described above. A variety of 
other periodic reporting comparisons, and statistical analysis should be designed 
to meet the needs of the project. 

The reporting of the analyse s will be made to several audiences. The primary 
reports will be made to the students being monitored and their teachers. These 
impacts on the students' learning will be greatest by putting the analysis of the 
results directly into the hands of the participants. The school administrators 
and the demonstration project staff in the state education department and at the 
University of Massachusetts will receive summaries of results reported to schools. 

The means of reporting must be efficient, rapid and comprehensive to complete 
the cycle of evaluation. The finest data will have no impact on the educational 
program of a school or district, if they are not readily available. The report 
of the analyses is printed by the computer in a form which can be easily read by 
teachers, students, and the demonstration project staff. 

Summative evaJ.uation of the demonstration project . Although CA11 monitoring 
is a system of evaluation, its usefulness in the natural setting to school per- 
sonnel and to students needs to be clearly documented. A systematic set of 
interview questionnaire items designed to elicit the opinions of students, teach- 
ers, and administrators will be used. Useful instruments are already available 
and are included in appendix C. 

Dissemination . Teachers, administrator, and state department personnel will 
accept and use CAM monitoring only if information about CAM is thoroughly dis- 
tributed. Specifically, the demonstration project must be systematically des- 
cribed and discussed throughout the state. The dissemination efforts must be 
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organized by at least one full-time project staff member with adequate resources. 
Dissemination will be directed towards several audiences and may call on outside 
consultants to develop the optimum strategy for certain audiences.. 

State level personnel will be one audience toward which the dissemination 
effort will be directed. Other audiences will Include personnel at schools not 
participating in the project, personnel teaching grade levels and subjects not 
using CAM monitoring in the schools participating in the project, and the general 
educational community. Each of these audiences must be informed, so that they 
will support one another. 

A vide variety of activities are planned for the dissemination. They will 
rely very heavily on personal visitation and participation in invitational, 
afternoon orientations to CAM in the demonstration schools. A newsletter to 
participant schools to report events which occur during the first year of CAM 
monitoring will help to cement the participants together. Articles written for 
professional journals will reach the larger educational community. 
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Possible Effects of CAM 



Effects for the Student 

A prina fncie benefit is that a high percentage of students enjoy the 
monitors and look forward to taking then. This is no snail gain when one con- 
siders the mixture of fear and loathing with which students often anticipate 
tests, A major task of the school is to create a positive attitude in the 
student with regard to education and the possibilities of grovrth in the con- 
temporary American school. Reasons for the positive feeling of students for 
the monitor are perhaps linked to their non-threatening character. The data 
analyses which are reported to the student allot; him to compete only with his 
past classmates. Profiles of personal achievement are reported to the student 
and not to his classmates. Intrinsic motivation is fostered. Secondly, the 
student becomes less obsessive about absolute scores and more concerend with 
his relative progress. He understands that he is not expected to score high" 
in the earlier phases of a curriculum or even in the later phases; that making 
a right or a \/rong answer is less important than the entire learning process 
which can give him a relative mastery over a period of time in a specific field. 

Also, though the student may not be explicitly aware of it, the learning 
opportunities which he is afforded are generally more congruent with his achieve- 
ment, his attitudes and his behavioral deficits. This is achieved through the 
on-going pretesting and increasingly delayed posttesting which are built into 
the CAM design. 

Effects for the Classroom Teacher 

One of the major advantages of CA11 is that it encourages the teacher to take 
a fresh look at an entire curriculum. In order to design a comprehensive monitor- 
ing program, it is necessary to 3;now with clarity and precision vrhat one’s learning 
objectives are. Teaching if facilitated when curriculum is seen as a series of 
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learning tasks that need careful consideration and precise articulation. 

From the data analysis which i3 reported to the teacher following each 
monitor, the teacher is enabled to make appropriate modifications in the pacing 
and the sequencing of learning tasks. When these tasks were less rigorously 
conceptualized and articulated, it is much more difficult to assess what class 
and individual achievement and therefore prescribe remediation. Self-corrective 
feedback suffered from the double fault of being defective and late. CAM ob- 
viates these difficulties because the objectives and items have been rigorously 
designed . 

CAM fosters a high degree of professionalism in the area of curriculum design 
and teaching method. It gives classroom teachers a criterion against which to 
assess their growth in a non-competitive way. Class achievement profiles provide 
the teacher with a personal record of teaching performance for self criticism. 

The workshops, in-service training sessions and professional meetings which 
will be held at appropriate intervals facilitates communication between classroom 
teachers who find that they have mutual concerns, and similar problems. 

There is a saving in time to the classroom teacher once the monitors have 
been designed. Absent are such time-consuming tasks as making up tests, correct- 
ing ' and evaluating them. 

Effects for the School 

The school as a whole is furnished with data processing and analysis which 
is more detailed, in-depth, and unbiased then might be furnished otherwise. The 
basis for compararive study, for curriculum and methods evaluation and for inter- 
school communication and interaction is furnished. CAM also fosters staff dif- 
ferentiation and inter-departmental interaction. This is because the implementation 
of CAM requires the cooperation of school faculty and staff at various levels and 
professional competencies. Schools do not have to allocate entire days to the 
administration of school-wide standardized tests. 
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E ffects for the State Department of Education 

As with those who function at the local level, personnel in the State 
Department of the Commonwealth can benefit from professional growth through 
the process of developing testing materials, curriculum designing and partici- 
pating in workshops. CAM, if administered on a state-wide level provides the 
state with a ready-make base for comparative studies of the quality of education 
and it *8 correlated with such variables as size of school, urban or rural cha- 
racter of the school, expenditures per pupil, and regional or local character 
of school. 

CAM, further, may create a network, which facilitates the cross -fertiliza- 
tion of ideas in many areas beside that of CAM itself. 
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BUDGET 



The planning and demonstration stages have been divided into separate 
budgets. The planning stage will produce (a) manuals which can be distributed 
throughout the State for general instructional purposes, (b) objectives and 
items for a base from which teachers could prepare additional materials, and 
(c) workshops which would be valuable professional opportunities by themselves. 
The planning stage could be funded and operate independent of the demonstra- 
tion stage although the converse would not be possible. 
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Budget 

Summary 



(1 January 1970 to 30 June 1971) 



Stage 


Date 

From To 




Category 




Amount 


I 


01/01/70 30/06/70 


1. 


Selection 




$ 3,040.0 






2. 


Objectives and Items 




11,686.8 






3. 


Training material 




7,808.0i 






4. 


Workshops 




13,292.0 


/ 




5. 


Programming 




5,085.0 


/ 




6, 


Summative Evaluation 




700.0 






7. 


Dissemination 




2,666.0< 






8, 


Indirect Costs 




1,520. 0( 






9. 


Local Contribution 


$18,500,00 








SUBTOTAL 




$45,797.8< 



01/07/70 


30/12/71 


1 . 


Administration 




$ 8,776.0< 






2. 


Workshops 




12.450.0C 






3. 


CAM Monitoring 




54.180.0C 






4. 


Summative Evaluation 




8.280.0C 






5. 


Dissemination 




13,474.0( 






6. 


Indirect Costs 




1,056.00 






7. 


Local Contribution 


$ 6,600.00 








SUBTOTAL 




$98,216.00 



TOTAL PROJECT 



$144,013.80 
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Bud pet 
Stap.e I 

(January 1970-30 June 1970) 



Item 



Terms 



Amount 



1. Selectinp Sample Population 



A. Salaries 

Director 1/4 X $1000. /mo. X 4 mo. = $1000 Local 
Associate Director 1/2 X $600. /mo. X 4 mo. $1200. 
Secretary to Assoc. Director 1/2 X $500. /no. X 4 no. 1000. 



B. Travel 
In-state for dir. 

and Assoc, dir. 

C. Equipment and supplies 
Typewriter rental 
Dic./trans. rental 
Supplies 

D. Other 
Telephone 
Office Space 

and furniture 



($ .08/ni. X 150 mi . / trip+$2 . 25food/trip)X 



2tp/wk. 


X 16 wk. 


($14. 25/trips) 


466. 


1/2 X $22 


. /mo . X 4 


tno. 


44. 


1/2 X $25 


./mo. X 4 


mo. 


50. 








100. 


$45. /mo. 


X 4 mo. 




180. 


(20 ft X 


30 ft) X : 


$ .50/sq. f t . /ino.X 




4 mo 


. = $1200 




Local 






TOTAL 


$3,040.00 
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Budget 
Stage I 
(continued) 



Item Terns Amount 



2. Writing Objectives and Items 



A. Salaries 

Coordinator 1/2 X $550. /mo. X A mo. $1100. 

Writers (one for each of AX $350. /mo. X A no. A600. 

four grades) 

Secretary $500, /no. X A mo. 2000. 

Clerical assistant $1.50 hr. X 120 hrs. 180. 

E. Travel 

Consultant $100. for airfare 100. 

$8.80 from airport 8.80 

$18.00 per diem for 5 days 90. 

C. Equipment 

Typewriter rental $22. /mo. X A mo. 08. 

Dic./trans. rental $25, /mo. X A mo. 100. 

D. Other 

Consultant for ULSIiA $100. /day X 5 days 500. 

Research Assistant $350. /mo. X A mo. 1A00. 

Pilot test duplication $.0A/copy X20 copies/pp. X 

A00 pages (i.e. s A00 items) 320. 

Optical scanning $. 05/sheet X 1 sheet/test X 

100 student/test A00. 

Computer time $5. /min. X 2 min. /test X CO tests 800. 



TOTAL 



$11,686.80 



*(CAIt will provide 2000 items and 2000 items will be written; all will be pretested; 
Local contribution = $6000.) 
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Budpet 
Stape I 
(continued) 



Item 



Terms 



Amount 



3. Preparinp traininp materials - four documents 



A. Salaries 
Associate Director 
Writer 

Secretary for writer 

B. Travel 
None 



1/4 X $600. /mo. X 4 mo. 
$600. /mo. X 4 mo. 

$500. /no. X 4 no. 



$ 600. 
2400. 
2000. 



C. Equipment 



Typewriter rental 


$22. /no. X 4 no. 


88. 


Dic./trans. rental 


$25. /no. X 4 no. 


100. 


Supplies 


$50, /no. X 4 mo. 


200. 


Other 


Research Assistant 


$350. /no. X 4 no. 


1400. 


Xerox 


$30. /mo. X 4 vo. 


120. 


Printinp 


$2. /copy X 100 copy/doc. X 4 doc. 


eoo. 


Graphic artist 


$50. /day X 2 days 


100. 



TOTAL .$7,808.00 
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Eudpet 
SLape I 
(continued) 



Item Terrs Amount 



4. Workshops (tiro) 



A. Salaries 
Director 
Assoc. Director 
Secretary to Assoc. Director 



1/4 X $1000. /mo. X 2 ro. = $50C. 
$600. /mo. X 2 mo. 

$500. /mo. X 2 mo. 



Local 

$ 1200 . 

1000. 



B. Travel 
Teachers 



$.08/mi. X 150 mi. /trip X 20 trips 48. 



C. Equipment and Supplies 
Typewriter rental 
Dic./tran6. rental 
Supplies 



$22. /mo. X 2 mo. 
$25. /mo. X 2 mo. 
$100. /mo. X 2 mo. 



44. 

50. 

200 . 



D. Other 

Teachers stipends 
Teachers room/board 
Constultant: statistics 



$175. /wk. X 2 vks./tchrs. X 40 tchr. 6000. 
$50. /wk. X 2 irks. /tchr. X 40 tchr. 4000. 
$150. /day X 5 days 750. 



TOTAL 



$13,292.00 
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Eudpet 
Stare 1 
(continued) 



Item 


Terns 


Account 


5. CPU r.onitorinp - conputer 


propranninr 




A. Travel 


Programmer 


California to )las8achu3c-tts 


$ 325. 


E . Other 


Computer propramFcr 


$100. /day X 20 dnyn 


2000. 


Conputer tine 


$300. /hr. X 0 hours 


2400. 


Computer programs 


C/i. contribution * $1G,00C 


Local 


Computer propranner 


$13. /day X 20 days 


360. 




TOTAL 


$5,085.00 



6. Surnnative evaluation 



A. Other 

:;estarch Assistant 



$350. /no. X 2 no. 



$ 700. 



TOTAL $ 700.00 




3 
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Budpet 
Stage I 
(continued) 









Item 


Terms 


Amount 


7. Dissemination 


A. Associate Director 


1/4 X $600. /mo. X 4 mo. 


$ 600. 


Secretary to assoc, dir. 


1/2 X $500. /mo. X 4 no. 


1000. 


Clerical 


$1. 50/hr. X 80 hours 


120. 


B. Travel 


Associate Director 


$. 08/mi. X 150 mi/trip X 






trip/no. X 4 ro. 


48. 


C. Equipment and supplies 


Typewriter rental 


$22. /no. X 4 no. 


88. 


Dic./trans. rental 


$25. /no. X 4 no. 


100. 


Supplies 




50. 


D. Other 


Telephone 


$20. /mo. X 4 no. 


80. 


Postape 


$20. /no. X 4 mo. 


80. 


Printing newsletter 


500 copies X 10 pap.es/copy 


300. 


Dissemination consultant 


$100. /day X 2 days 


200. 




TOTAL 


$2,666.00 


8. Indirect costs 


83! X $19,000.00, i.c., funds 






subject to indirect costs 


$1,520.00 



9. Local contribution 



$18,500.00 



Local 
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TOTAL STAGE I 



$45,797.80 



Budget 

Stage II 
(continued) 



I ten 


Terms 




Amount 


1. Administration 








A. Salaries 


Director 


1/4 X $1000. /ro. X 12 mo. 




Local 


Associate Director 


1/2 X $ 600. /mo. X 12 no. 




$3600. 


Secretary to Assoc. Dir. 


1/2 X $ 500. ho. X 12 mo. 




300C. 


B. Travel 


($.08/ni X 150 ri./trip + 
4 trips/no. X 12 ro. 


$2. 25/trip)X 


684. 


C. Equipment and supplies 


Typewriter rental 


1/2 X $22. /no. X 12 mo. 




142. 


Bic./trans. rental 


1/2 X $25. /mo. X 12 no. 




150. 


Supplies 


$20. /no. X 12 ro. 




240. 


D. Other 


Telephone 


$1C. A'k. X 4 wk. /no . X 12 


tro* 


480. 


Postape 


$10. /no. X 12 no. 




120. 


Xerox 


$30. /mo. X 12 ro. 




360. 


Office space 


(20 ft. x 3C ft.) X $. 50/sq.ft. /mo. 
X 12 mo. » $3600 


Local 



TOTAL $8,776.00 



2. In service Workshops 
A. Travel 



Teachers to workshops 


$.08/ai. X 150 mi. /trip X 






20 trips /workshop X 5 weeks 


$1200. 


B. Supplies 


Supplies 


$100. /workshop X 5 workshops 


500. 


C. Other 


Teacher's Stipends 


$5C,/tch. X 40 tchr. /workshop X 




• 


5 workshops 


10000. 


Consultant 


$150. /consultant/workshop X 5 workshops 


750. 



TOTAL 



$12,450.00 
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L«udpet 
Stape II 
(continued) 



Item 


Terns 


Amount 


3. CAH nonitorinp expenses 
A, Travel 


Supervisor trip to 


$.08/r.i. 7. 15C r-i./trip X 


$1200. 


demonstration schools 


10 trips/ro. X 10 ix). 




Is. Other 


Designing CAK tests 


$100./prade X 8 grades 


800. 


Printing tests and objectives 


$. 20/copy X 100 copics/tcst X 






20 test/f rade X C grades 


3200. 


Keypunching 


$4. /hr. X 4C hr. /irk. X 8 wk. 


1280. 


Periodic Processing 


$. 50/test X 10 tests/student X 




($5 . 00/student) 


300 student /school X 20 schools 


30000. 


Final Processing 


$.5G/ntudent X 300 st. /school X 






20 each 


3000. 


Research Assistant 


(achiever. ent profiles and iter* 
analysis) 

$350. /ro. X 10 no. /asst. X 






1 asst./5 schools X 20 schools 


14000. 


Telephone 


$10./tik. X 4 vks./ro. X 10 t». 


400. 


Postage 


$30. /no. X 10 no. 


300. 




TOTAL 


$54, 180 .'66 



4. Sunnative evaluation 
A. Travel 


research Assistant 


$.Ce/ni. X 150 nl./trlp X 






2 tp./sch. X 20 schools 


480. 


B. Other 


Research Assistant 


$350. /do. X 10 nc. 


3500. 


Consultant 


$150. /day 7. 2 days 


300. 


Supplies 


Questionnaire 


(trritinp, typing, pretestinp, admin.) 




for students 


12.000 copies (pre and post) 


700. 


for teachers 


CO copies (pre and no3t) 


300. 


Optical scanning 


$.C5/sheol X 12.0CO sheets 


600. 


Computer tine 


$300. /hr. X 8 hr. 


2400. 



TOTAL 



$6,260.00 
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Budget 
Stage II 
(continued) 



Item 


Terns 


Amount 


5. Dissemination 
A. Salaries 


Associate Director 


1/2 X $600. /no. X 12 ro. 


$3600. 


Secretary to Assoc. Dir. 


1/2 C $3C0./r.o. C 12 no. 


3000. 


E. Travel 


Director 


$100/trip X 2 trips to Washington. 
$60/trip X 2 trips to N.Y.C. 


D.C.+ 

320. 


Associate Director 


$100/trip C 1 trip to Washington, I.C.+ 

$C0/trip X 1 trip to U.Y.G.d 

$00/tri]» Z 1 trip to LPDC in Pittsburgh+ 

($. 08/mi. X 150 mi. /trip + $2.25/trip)X 

2 trip3/mo. X 12 ro. 342. 


C. Equipment and supplies 


Typev>riter rental 


1/2 X $22. /no. X 1? mo. 


142. 


Dic./trans. rental 


1/2 X $25. /ro. X 12 no. 


150. 


Supplies 


$20. /r.o. X 12 ro. 


480. 


D. Other 


Telephone 


$40. /r.o . X 1 2 to) . 


960. 


Postage 


$20. /no. X 12 mo. 


480. 


Printing newsletter 


$500. /issue X 5 issue 


2500. 


Dissemination consultant 


$100. /day X 5 days 


500. 


Final Report 


500 copies of ICO pages 


1000. 




TOTAL 


$13,474.00 


.6. Indirect cocts 


83 X $13,200.00 i.e. 

funds subject to Indirect costs 


$1,056.00 



7* Local contributions $6600 



Local 



TOTAL STAG! II ${6,216, CO 



O 
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Appendix A 



Technical Memoranda 
and 

Blbllopraphy 
of the Project for 

Comprehensive Achievement Konltorinp 



Technical Memoranda 



Number Title 


Author 


AR-1 


First Annual Report 


Gorth 


TM-2 


Description of courses monitored by Project CAM 


Gorth & Popejoy 


TM-3 


Monltorir " Tchedules developed for research by 
Project CAM. 


Corth, Stroud, 
& Knight 


TM-4 


The relation of repeated, comprehensive pretest- 
ing and student's achievement 


Gorth, Allen, 
Popejoy, & Stroud 


TM-5 


A comparison of comprehensive versus unit pretest- 
ing and student's achievement 


Gorth, Allen, 
Popejoy, Stroud 


TM-6 


The evaluation of item performance in an item sam- 
pling case 


Lindeman, Gorth, 
& Allen 


TM-7 


Computer-Based, insturctional-testing data bank 


Popejoy, Gorth, 
Grayson & Stroud 


TM-8 


Separate />nalyses of regression 


Stroud 6 Gorth 


TM-9 


Educational Innovations monitored by Project CAM 


Gorth 


TH-10 


Longitudinal comprehensive achievement monitoring 
in science education 


Gorth & Allen 


TM-11 


A computer program to evaluate item performance by 
internal and external criteria in a longitudinal 


Gorth, Grayson 
& Lindeman 


TM-12 


A computer program to tabulate performance profiles 
of longitudinal performance testing using item 
sampling 


Gorth, Grayson, 
& Stroud 


TM-13 


The project CAM data blak for 1967-1968 


Gorth 


TH-14 


A computer program to compose and print teats for 
instructional testing using item sampling 


Gorth 


TM-15 


Investigating a linear model of learning in ninth 
grade algebra 


Stroud & Gorth 


TH-16 


Analysis of the Project CAM data for 1967-1968 


Corth & Plnsky 


TH-1? 


Konltoring schedules developed for research; 1968- 
1969 


PinsV/ & Gorth 


TM-18 


Demographic, aptitude, & attitude surveys of the 
students, teachers, and schools in Project CAM 


Gorth & Plnsky 


AA-2 


Second Annual Report to the Charles F. Kettering 
Foundation 


Allen & Gorth 


These 


reports and further information may be obtained by contacting the Project < 



or writing to William P. Gorth, Director, Project CAH, School of Education, University 
of Kaaaachusetta, Massachusetts 01002. 
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Technical Memoranda 



Number 


Title 


Author 


TM-19 


Issues in comparinp achievement patterns illustrated 
with data from a naval traininp propran 


Burke & Gorth 


Til- 20 


CA1' described for state level evaluation of urban edu- 
cation projects 


Gorth & Nightman 


TM-21 


Descriptive analysis of HS420, eleventh prade algebra, 
first semester 


Pinsky & Gorth 


Til— 22 


Descriptive analysis of KA442, one semester eleventh 
and twelfth prade trigonometry 


P insky & Gorth 



Professional Meetings 

Gorth, W. (Organizer) Comprehensive Achievement Monitorinp. Symposium presented 
at the meeting of the American Educational Research Association, Los Anpeles, 
February, 1969. 

Gorth, tJ. and Allen, D. A new deafen for evaluation in mathematics education. 
Paper to be presented at the Sfoetinp of the National Council of Teachers 
of Mathematics, Minneapolis, April, 1969. 

Llndeman, R. , and Gorth, V. and Allen D. Iten analysis in an item sarnplinp case. 
Paper presented at the annual r.eetinp of the National Council of Measurement 
in Education, Los Anpeles, February, 1969. 



Forthcomlnp, 

Gorth, U. (Grpanizer) Desipnitip instructional systems with longitudinal testinp 
usinp item samplinp techniques. Symposium to be presented at the American 
Educational Research Association, Minneapolis, March, 1970. 



These reports ar.d further information my be obtained by contsctinp the Project 
CAM staff or writing. to WilliAn P. Gorth, Director, Project CAM, School of Edu- 
cation, University of Massachusetts, Anherst , Massachusetts, C1002. 



Workin g Paper s 



Humber Titl e 

WP-1 The classroom teacher's manual for Comprehensive 
Achievement Monitoring 

WP-3 Anxiety , achievement, and Project CAli 



A uthor 

Gorth 

Gorth, Paulson, 
& Sieber 



W P-4 Data processing for achievement monitoring 



Gorth & Pinsky 



VP-5 Seven premises in search of a conclusion OP. the Yamashita 

game 

VHP-6 An instructional management system designed with Pinsky 

Comprehensive Achievement Monitoring (/IRA, 69) 

VP-7 Instructional objectives, achievement monitorinp, Gorth 
and learning, what, how, why, and where of Com- 
prehensive Achievement Monitoring (AEFA, 69) 



WP-3 A new design for evaluation in mathematics educa- Gorth & Allen 
tion- lonpitudinal Comprehensive Achievement 
Monitorinp (KCTW, 69) 



P ublication s 

Gorth & Grayson. A program to compose and print tests for instruction*. 1 
testing using item sampling. Ed ucational n ti d Psychological Measure- 
ment , 1969, 29, 173-174. 

Gorth, Grayson, & Lindeman. A computer program, to evaluate item performance 
by internal and external criteria in a longitudinal testinp program 
u9inp item sannlinp. Educational and Psychological Measurement, 1969, 
29, 181-183. 

Gorth, Grayson, 6 Stroud. A computer proprar to tabluate and plot achieve- 
ment profiles of longitudinal achievement testing using item sampling.. 
Educationa l and Psychological Measurement , 1969, 29, 179-180. 

Gorth, Allen, Popejoy, & Stroud. A comparison of comprehensive versus unit 
pretesting a9 related to student achievement. Psychol o gy in the School , 
1969, 6, 391-393. 

Gorth, Grayson, Topejoy, 6 Stroud. / tape-based data bank for educational 
research or instructional testing using lonpitudinal item sampling. 
Educational and Psychological Measurement . 1969, 29, 175-177. 

Vightoan 6 Gorth. CAM: the new look in classroom testing. Trend , .969, 

Spring, 58-57. 



These reports and futther information may be obtained by contacting the Project 
CAM staff or writing to V’lilian T. Gorth, Director, Itoject CAM, School of Edu- 
ction, University of Massachusetts, Amherst, Massachusetts, 01002. 



Appendix B 



Participating Schools With 
Comprehensive Achievement Ifonitorinp 



Hopkins High School 

Ray Weidner, Director 
Chuck Thiele 
Janes Whitney 

Andrew Jackson Hlph School 
Roy Carlson, Principal 

Jean Stromquist, Chairman 

Robert Christiansen 

Richard Clark 

David LarseJl 

Janes Morton 

Donald Ronine 

Jeanne Steed 

Kailua high School 

Hilton Detiello, Principal 
Clare Callatt 
June Yaciashltr. 

John Marshall Hiph School 

Dr. Gaynor Petriquin, Principal 
Max Lane 
Ray O'Dell 

Lincoln- Sudbury H.iph School 
Alex Marshall 

Ballston SPA 

Gerald Murphy 

Board of Cooperative educational 
Service of Westchester County 
V'alter Goodman 

Levittown Public Schools 

Louis Pucci, Principle Investipator 
Jack Robbins, Director 



Hopkins, Minnesota 



Portland, Orcron 



Kailua, Hava i i 



Portland, Crtron 



Lincoln;. Massachusetts 



Ballston SPA, Mew York 



Yorktown Leiphts, Lew York 



Levittown, Dew York 
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Appendix C 



Suroatlve Evaluation 
Student questionnaire 



and 



Teacher Interview 




Project CAM 

Questionnaire of Student Opinion 



(last) PRINT NAME (first) COURSE SCHOOL 

This sheet contains statements of opinion. This is NOT a test. 

We Just want to know what you really think about CAM testing. 

w 

CIRCLE THE ONE LETTER BEST DESCRIBING YOUR OPINION. ffc 

<4 

Example: g 

I enjoy work as much as play, (If you slightly agree, 

you would circle C.) A 

1. There is usually enough time to answer all of the CAM test 

questions. A 

2. There are too many CAM tests given in this course. A 

3. A lot can be learned by just taking the CAM tests. A 

4. The CAM tests don't help me to know what I should study 

next . A 

5. It would help me to learn more if there were CAM tests in 

my other classes. A 

6. We should not be asked to answer questions on things that 

have not been taught in class. A 

7. The computer report of my CAM tests helps me figure out 

what material I haven't learned very well. A 

8. There aren't many things in this subject that are 

interesting. A 

9. The CAM tests are not fair measures of what I know. A 

10. Tests like CAM tests should be used in most of my other 

courses. A 

11. Most of the tl*e 1 can't understand what the CAM test items 

really mean. A 

12. The CAM tests help to point out material we will have to 

learn next. A 

13. I like to take CAM tests. A 

14. I get my CAM test results back too late to do any good. A 

15. If we have to have tests, they should be like the CAM tests. A 

16. Most of the other students in the course like to take CAM 

tests. A 

17. It would be better to quit CAM testing and have only tests 

like we used to have. A 

18. The CAM tests don't tell what I really have learned in 

this course. A 

19. I could not tell from the CAM tests if I was improving in 

the couree, A 

20* The CAM tests are good because the teacher can find out 

what the class needs help on. A 

O 
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C D 
C D 

C D 

C D 

C D 

C D 

C D 



Slightly 



Please answer these questions as honestly and as thoroughly as you 
can. Your answers will help to Improve the CAM testing program. 

(a) Write down several specific advantages that you can see in 
using CAM testing aid reporting. Be specific . Particularly 
mention things which you like about CAM testing or which you 
£eel may be better in CAM testing than regular testing. 



(b) Write down several specific disadvantages that you can see in 
using CAM testing and reporting. Be specific . Particularly 
mention things which you dislike about CAM testing or which you 
feel may be better in CAM testing than regular testing. 



(c) What do you do with the CAM computer reports of your CAM tests? 



(d) What improvements would you like to see in CAM testing or report- 
ing? Be specific . 



(e) Do you feel your grades should be based on your CAM test results 
and why? 



(f) How hard do you try when you take the CAM tests? Do you always 
try this hard? 




Project CAH 



1.00 

2.00 

2.10 

2.20 

2.40 

2.41 

2.42 
2.60 

2.62 

2.80 

2.82 

4.00 

4.10 

4.20 

4.21 
4.30 

4.41 

4.42 

4.60 

4.62 

6.00 

6.10 

6.21 



Post Course Interview of Teachero 
Description of Structure. Content, and Ervro Ilshat of Couroe 
Preparing Performance Object ivae 

How did you prepare objectives, e.g., write or select? 

What is your general attitude toward defining the goals of a course 
in performance objectives and why? 

When did you prepare the objectives for your course? 

How much time did you spend the first time you prepared objectives for 
your complete course. 6»g*. total working days or weeks for all people 
involved in course? 

If you rewrote the objectives, how much time did the rewrite take? 

Are you able to modify the objectives during the course and then 
reteach them? 

On what basis did you modify the objectives, l.e*. where did you get 
information suggesting they should be changed? 

Did you add or t'-hte objectives for ymr course? 

On what basis did you add or delete? 

Preparing Questions for CAM 

What is your general attitude toward preparing questions and how did 
you prepare them, e.g*. write or select? 

When did you prepare questions? 

How much time did you spend preparing questions? 

What kinds of difficulties did you encounter when you were prepsring 
questions? 

Before using CAM did you reuse questions from year to year? 

Before using CAM did you reuse questions during the year, e.g., test- 
retest? 

Do you modify questions from regular or CAM testing systematically? 

On what basic do you modify questions? 

Test Preparation 

How did you prepare teste for CAM? 

When did y u prepare the usual tests for your course? 





Project GAM 2 

Post Course Interview of Teachers 


6.22 


When did you prepare GAM teste? 


8.00 


Student Scheduling 


8.11 


How was the CAM testing schedule prepared, i.e., for time and student 
assignnent? 


8.20 


How much time did it take to schedule students? 


10.00 


Administration of GAM Tests 


10.11 


How did you administer CAM te 3 ts, i.e., what were the mechanics of 
getting the correct test to the correct student? 


10.12 


Compare the time and effort between CAM and regular test administration. 


10.13 


What is a good frequency? 


10.21 

10.22 


What are the difficulties in CAM test administration? 
What changes could be made to lessen these difficult iea? 


10.30 


How would improved CAM test administration compare with regular class- 
room testing? 


11.00 


Scoring and Analysis of CAM Data 


11.10 


Who scored and analyzed the CAM tests? 


11.20 


What scoring or analyses did you perform? 


12.00 


Teacher's Perceptions of Student's Response to CAM 


12.10 


What is the students' general attitude toward CAM testing? 


12.20 


What do you think they like about CAM testing? 


12.30 


What do they dislike about CAM testing? 


12.40 


How consistent are their efforts while taking CAM tests, i.e., do they 
sometimes try and sometimes not? 


12.51 


What per cent do you think really try hard? 


12.52 

12.60 


What per cent do you think do not try at all? 
How would you suggest improving student efforts? 


12.61 


What effect would counting CAM testing in grades have on student 
interest. 


12.70 


How do the students use the feedback? 


12.71 

12.72 
12.80 


What per cent of the students use the feedback? 

Do the students each have their own copy of the objectives? 
How would you suggest improving student feedback? 


O 
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Project CAM 

Poa t Course Interview of Teachers 



3 



12.91 How much of a learning device is the CAM testing? 

12.92 How much of a learning device is the CAM feedback? 

14.00 Teacher *8 Perceptions of Congruence of Questions and Course 

14.20 What aspects of student performance on the objectives are measured 
adequatdy by the questions? 

14.30 What aspects of the student's performance on the ob> ctlves are not 
measured? 

14.40 What objectives are not measured? 

14.50 At what level of difficulty are the questions written, e.g., 

mastery level where everyone should be able to answer them by the 
end of the course? 

16.00 Teacher's Attitudes Toward CAM 

16.10 What are the good aspects of CAM monitoring? 

16.20 What are the bad aspects of CAM monitoring? 

16.30 How do you feel the monitoring should be improved? 

16.40 How do you use' the feedback and with what per' cant of the students? 

16.41 Hov would you use it? 

16.42 Do you have enough time to use the feedback? 

16.60 Do you use it for grading? 

16*62 What per cent of the final grade in the course depends on the CAM 

testing results? 

18.00 Dissemination About Project CAM 

18.10 With whom have you discussed CAM and what were their reactions (get 
number of people, the£r occupations)? 

18.20 To whom have you made presentations about Project CAM? 



O 
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